Harden databento ingest: progress/timing, durable + reconcilable manifest, windowed n.1 stats#40
Open
mspinola wants to merge 3 commits into
Open
Harden databento ingest: progress/timing, durable + reconcilable manifest, windowed n.1 stats#40mspinola wants to merge 3 commits into
mspinola wants to merge 3 commits into
Conversation
…fest, windowed n.1 stats Operational hardening for the databento ingest, driven by a ~24h backfill. - Progress + timing: per-symbol [i/N] with run-elapsed, and per-fetch duration (reveals the statistics schema as the bottleneck). - Durable manifest: persisted after EVERY fetch instead of only at the end, so an interrupted run resumes instead of re-downloading. - reconcile_manifest() + 'cotdata-update --reconcile-databento': rebuild the manifest from the raw parquets on disk (recovery for runs that predate the durable write). No API. - Resume bug fix: skip when start >= end. databento requires start < end, so an already-current symbol 422'd (data_time_range_start_on_or_after_end) and falsely reported failure — which would make every steady-state run exit non-zero. - 'cotdata-update --windowed-n1-stats [DAYS]' (default 3): fetch the n.1 statistics schema only in windows around roll dates (settlement is only needed at rolls). Validated on downloaded data: roll gaps IDENTICAL to the full pull, 67-89% fewer n.1 rows (~30-35% off total runtime), zero accuracy loss. Watermark resume retries transient window failures rather than silently dropping a roll's gap. 113 tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The historical streaming API (timeseries.get_range) throws read timeouts, aborted connections, and 'Response ended prematurely' often on large from-inception pulls. _fetch now retries 3x with linear backoff so these self-heal within the run instead of dropping the fetch (which loses a symbol/roll-window until the next re-run). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Streaming timeseries.get_range times out on from-inception statistics (read timeouts, dropped streams). ingest_batch() submits one databento BATCH job per (feed, schema) covering all symbols still needing it, waits, downloads the CSV files, and parses them into the SAME raw store + manifest — so build() and reconcile() are unchanged. Statistics are fetched full (batch handles the volume; no windowing needed). Routed via 'cotdata-update --ingest-databento --batch'. The CSV parsing follows the proven dormant run_batch_backfill pattern (groupby symbol, instrument_id for rolls). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Operational hardening for the databento ingest, driven by a ~24-hour backfill run. All on top of the merged two-stage producer.
Progress + timing
Per-symbol
[i/N]with run-elapsed, and per-fetch duration — which makes thestatisticsschema visibly the bottleneck (millions of rows) vsohlcv-1d(seconds).Durable manifest
The ingest manifest is now persisted after every fetch instead of only at the end, so an interrupted run resumes where it left off instead of re-downloading. Plus
reconcile_manifest()/cotdata-update --reconcile-databentoto rebuild the manifest from the raw parquets already on disk (recovery for a run that predates the durable write; reads local files only, no API).Resume bug fix (422)
databento's time range is half-open (
start < end), so an already-current symbol computedstart == endand 422'd (data_time_range_start_on_or_after_end), falsely counting as a failure. That would make every steady-state (nothing-new) run exit non-zero — e.g. the nightly cron false-alarming. Guard changed from>to>=.Windowed n.1 statistics (
--windowed-n1-stats [DAYS], default 3)n.1 settlement is only needed at roll dates, so fetch the n.1
statisticsschema only in ±DAYS windows around each roll (found from the on-disk n.0instrument_id) instead of full history.Validated on already-downloaded data — roll gaps are identical to the full pull, at a fraction of the download:
Since n.1 statistics is ~30–45% of the total, that's ~30–35% off total runtime with no back-adjustment accuracy loss. The full n.0 statistics (daily settlement + OI) and n.1 ohlcv are unchanged. A watermark advances the resume point only through the last contiguous successful window, so a transient failure is retried rather than silently leaving a roll's gap unmeasured. Trades bytes for call count (one small n.1 query per roll).
113 tests green (incl. a full-vs-windowed identical-back-adjustment test, a
start == endregression, and a manifest-reconcile test).🤖 Generated with Claude Code